fix(cold-store): tolerate torn trailing line in parse_jsonl - #80
Merged
Merged
Conversation
Under concurrent producer write + reader scan (the e2e harness shape), the cold-fallback reader saw a part-file whose final line was a partial JSON record and errored the whole scan, manifesting as 10 s cold-fallback timeouts on `sum(http_requests_total)` in E0 (PR #261, 34/300 replay queries). Tolerate the torn last line specifically: if a parse error hits the *final* line *and* the blob has no terminating newline (the producer-mid-flush shape per design-sketch-db.md §5.2), warn + drop instead of failing the part. Mid-file parse errors still hard-error — that signals real corruption, not a concurrent write. Threads `Option<&Path>` through a new `parse_jsonl_at` helper so the warn log includes the offending file + a 200-char line preview; existing `parse_jsonl(bytes, start, end)` callers (`local_fs.rs`) unchanged. Pins both shapes in tests: - parse_jsonl_ignores_torn_trailing_line - parse_jsonl_errors_on_mid_file_corruption Unblocks PROGRESS.md "Outstanding follow-ups #5" and the E0 exit-criterion (1) cold-routed query family. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parse_jsonlnow drops a malformed trailing line with a warn log when the blob has no terminating newline (the producer-mid-flush shape from design-sketch-db.md §5.2). Mid-file malformed lines still hard-error so real corruption surfaces.Option<&Path>through a newparse_jsonl_athelper so the warn log includes the file path + 200-char line preview. Existingparse_jsonl(bytes, start, end)callers (local_fs.rs) are unchanged.sum(http_requests_total)because of this).Test plan
parse_jsonl_ignores_torn_trailing_line— 3 valid lines + partial 4th, asserts 3 records + no error.parse_jsonl_errors_on_mid_file_corruption— valid / not-json / valid (newline-terminated), assertsMalformed("line 2: …").parse_jsonl_filters_range,parse_jsonl_malformed_errs,prefix_shape,hour_prefixes_*, and alllocal_fs::tests::scan_*still green.cargo test --release -p query_engine_rust --lib drivers::query::fallback::cold_store— 11 passed, 0 failed.cargo clippy --release -p query_engine_rust --all-targets -- -D warnings— pre-existing failures insimple_engine.rs/count_sketch_accumulator.rs/otel.rsunchanged; no new lints introduced by this change.🤖 Generated with Claude Code